Server side hotwording

ABSTRACT

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for detecting hotwords using a server. One of the methods includes receiving an audio signal encoding one or more utterances including a first utterance; determining whether at least a portion of the first utterance satisfies a first threshold of being at least a portion of a key phrase; in response to determining that at least the portion of the first utterance satisfies the first threshold of being at least a portion of a key phrase, sending the audio signal to a server system that determines whether the first utterance satisfies a second threshold of being the key phrase, the second threshold being more restrictive than the first threshold; and receiving tagged text data representing the one or more utterances encoded in the audio signal when the server system determines that the first utterance satisfies the second threshold.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application is a continuation of, and claims priority under35 U.S.C. § 120 from, U.S. patent application Ser. No. 16/392,829, filedon Apr. 24, 2019, which is a continuation of U.S. patent applicationSer. No. 15/432,358, filed on Feb. 14, 2017. The disclosures of theseprior applications are considered part of the disclosure of thisapplication and are hereby incorporated by reference in theirentireties.

BACKGROUND

Automatic speech recognition is one technology that is used in mobiledevices among other types of devices. One task that is a common goal forthis technology is to be able to use voice commands to wake up a deviceand have basic spoken interactions with the device. For example, it maybe desirable for the device to recognize a “hotword” that signals thatthe device should activate when the device is in a sleep state.

SUMMARY

A system may use two thresholds to determine whether a user spoke a keyphrase. A client device, included in the system, uses the first, lowerthreshold to determine whether a portion of words spoken by the user arethe same as a portion of the key phrase. For instance, when the keyphrase is “okay google,” the client device may use the first, lowerthreshold to determine whether the user spoke “okay” or “okay g” or“okay google.” When the client device determines that the portion of thewords spoken by the user are the same as a portion of the key phrase,the client device sends data for the words to a server. The server usesa second, higher threshold to determine whether the words spoken by theuser are the same as the key phrase. The server analyzes the entire keyphrase to determine whether the user spoke the key phrase. When theserver determines that the key phrase is included in the words, theserver may parse other words spoken by the user to generate data for anaction that the client device should perform.

In some implementations, a client device may receive an audio signalthat encodes one or more utterances. The client device uses a firstthreshold to determine whether one or more first utterances encoded atthe beginning of the audio signal satisfy a first threshold of being akey phrase. The client device may analyze a portion of an utterance, asingle utterance from the one or more first utterances when the keyphrase includes multiple words, or both.

When the client device determines that the one or more first utterancessatisfy the first threshold of being a key phrase, the client devicesends the audio signal to a speech recognition system, e.g., included ona server separate from the client device, for additional analysis. Thespeech recognition system receives the audio signal. The speechrecognition system analyzes the one or more first utterances todetermine whether the one or more first utterances satisfy a secondthreshold of being the key phrase. The second threshold is morerestrictive than the first threshold, e.g., the first threshold is lessaccurate or lower than the second threshold. For instance, when thefirst threshold and the second threshold are both likelihoods, the firstthreshold, e.g., fifty percent, is a lower likelihood than the secondthreshold, e.g., seventy-five or ninety percent. For the one or morefirst utterances to satisfy the first threshold or the second thresholdof being a key phrase, a corresponding system may determine that thelikelihood of the one or more first utterances being the key phrase isgreater than, or greater than or equal to, the respective threshold.

The speech recognition system receives, from the client device, data forthe entire audio signal including the one or more first utterances sothat the speech recognition system can analyze all of the data includedin the audio signal. For example, to reduce the possibility of thespeech recognition system receiving data for an audio signal thatincludes a partial utterance at the beginning of the audio signal, toimprove the speech recognition analysis by the speech recognitionsystem, or both, the speech recognition system receives the entire audiosignal from the client device when the client device determines that atleast a portion of the one or more first utterances satisfy the firstthreshold of being the key phrase.

In some implementations, the client device may analyze a prefix or aportion of one of the first utterances. For instance, when the keyphrase is “Okay Google”, the client device may determine that the one ormore first utterances encode “Okay G” or “Okay” without analyzing all ofthe second utterance and, in response, send the audio signal to thespeech recognition system.

In some implementations, the client device may send, with the data forthe audio signal and to the speech recognition system, data for the keyphrase. The data for the key phrase may be text representing the keyphrase, or an identifier, e.g., for the client device, which the speechrecognition system may use to determine the key phrase. The speechrecognition system may use the data for the key phrase to determinewhether the one or more first utterances included in the audio signalsatisfy the second threshold of being the key phrase.

In general, one innovative aspect of the subject matter described inthis specification can be embodied in methods that include the actionsof receiving an audio signal encoding one or more utterances including afirst utterance; determining whether at least a portion of the firstutterance satisfies a first threshold of being at least a portion of akey phrase; in response to determining that at least the portion of thefirst utterance satisfies the first threshold of being at least aportion of a key phrase, sending the audio signal to a server systemthat determines whether the first utterance satisfies a second thresholdof being the key phrase, the second threshold being more restrictivethan the first threshold; and receiving, from the server system, taggedtext data representing the one or more utterances encoded in the audiosignal when the server system determines that the first utterancesatisfies the second threshold. Other embodiments of this aspect includecorresponding computer systems, apparatus, and computer programsrecorded on one or more computer storage devices, each configured toperform the actions of the methods. A system of one or more computerscan be configured to perform particular operations or actions by virtueof having software, firmware, hardware, or a combination of theminstalled on the system that in operation causes or cause the system toperform the actions. One or more computer programs can be configured toperform particular operations or actions by virtue of includinginstructions that, when executed by data processing apparatus, cause theapparatus to perform the actions.

In general, one innovative aspect of the subject matter described inthis specification can be embodied in methods that include the actionsof receiving, from a client device, an audio signal encoding one or moreutterances including one or more first utterances for which the clientdevice determined that at least a portion of the one or more firstutterances satisfies a first threshold of being at least a portion of akey phrase; determining whether the one or more first utterances satisfya second threshold of being at least a portion of the key phrase, thesecond threshold more restrictive than the first threshold; and sending,to the client device, a result of determining whether the one or morefirst utterances satisfy the second threshold of being the key phrase.Other embodiments of this aspect include corresponding computer systems,apparatus, and computer programs recorded on one or more computerstorage devices, each configured to perform the actions of the methods.A system of one or more computers can be configured to performparticular operations or actions by virtue of having software, firmware,hardware, or a combination of them installed on the system that inoperation causes or cause the system to perform the actions. One or morecomputer programs can be configured to perform particular operations oractions by virtue of including instructions that, when executed by dataprocessing apparatus, cause the apparatus to perform the actions.

The foregoing and other embodiments can each optionally include one ormore of the following features, alone or in combination. The method mayinclude performing an action using the tagged text data subsequent toreceiving, from the server system, the tagged text data representing theone or more utterances encoded in the audio signal when the serversystem determines that the first utterance satisfies the secondthreshold. The one or more utterances may include two or moreutterances, the first utterance encoded prior to the other utterancesfrom the two or more utterances in the audio signal. Performing theaction using the tagged text data may include performing an action usingthe tagged text data for the one or more utterances encoded in the audiosignal after the first utterance. Determining whether at least a portionof the first utterance satisfies the first threshold of being at least aportion of the key phrase may include determining whether at least aportion of the first utterance satisfies the first threshold of being atleast a portion of the key phrase that includes two or more words.

In some implementations, the method may include receiving a second audiosignal encoding one or more second utterances including a secondutterance; determining whether at least a portion of the secondutterance satisfies the first threshold of being at least a portion of akey phrase; and in response to determining that at least the portion ofthe second utterance does not satisfy the first threshold of being atleast a portion of a key phrase, discarding the second audio signal. Themethod may include determining to not perform an action using data fromthe second audio signal in response to determining that at least theportion of the second utterance does not satisfy the first threshold ofbeing at least a portion of a key phrase. Determining whether at least aportion of the first utterance satisfies the first threshold of being akey phrase may include determining whether at least a portion of thefirst utterance satisfies a first likelihood of being at least a portionof a key phrase.

In some implementations, sending, to the client device, the result ofdetermining whether the one or more first utterances satisfy the secondthreshold of being the key phrase may include sending, to the clientdevice, data indicating that the key phrase is not likely included inthe audio signal in response to determining that the one or more firstutterances do not satisfy the second threshold of being the key phrase.Sending, to the client device, the result of determining whether the oneor more first utterances satisfy the second threshold of being the keyphrase may include sending, to the client device, data for the audiosignal in response to determining that the one or more first utterancessatisfy the second threshold of being the key phrase. Sending, to theclient device, data for the audio signal in response to determining thatthe one or more first utterances satisfy the second threshold of beingthe key phrase may include sending, to the client device, tagged textdata representing the one or more utterances encoded in the audiosignal. The method may include analyzing the entire audio signal todetermine first data for each of the one or more utterances. Sending, tothe client device, the data for the audio signal in response todetermining that the one or more first utterances satisfy the secondthreshold of being the key phrase may include sending, to the clientdevice, the first data for the audio signal in response to determiningthat the one or more first utterances satisfy the second threshold ofbeing the key phrase.

In some implementations, determining whether the one or more firstutterances satisfy the second threshold of being the key phrase mayinclude determining, using a language model, whether the one or morefirst utterances satisfy the second threshold of being the key phrase.The method may include customizing the language model for the key phraseprior to determining, using the language model, whether the one or morefirst utterances satisfy the second threshold of being the key phrase.The method may include receiving text identifying the key phrase.Customizing the language model for the key phrase may includecustomizing the language model for the key phrase using the textidentifying the key phrase. The method may include receiving anidentifier, and determining, using the identifier, key phrase data forthe key phrase. Customizing the language model for the key phrase mayinclude customizing the language model for the key phrase using the keyphrase data. Determining, using the language model, whether the one ormore first utterances satisfy the second threshold of being the keyphrase may include determining, using the language model and an acousticmodel, whether the one or more first utterances satisfy the secondthreshold of being the key phrase. Determining, using the language modeland the acoustic model, whether the one or more first utterances satisfythe second threshold of being the key phrase may include providing datafor the one or more first utterances to the language model to cause thelanguage model generate a first output; providing data for the one ormore first utterances to the acoustic model to cause the acoustic modelto generate a second output; combining the first output and the secondoutput to generate a combined output, and determining, using thecombined output, whether the one or more first utterances satisfy thesecond threshold of being the key phrase. The method may includeselecting the language model for a default key phrase. The method mayinclude determining whether to use the default key phrase.

The subject matter described in this specification can be implemented inparticular embodiments and may result in one or more of the followingadvantages. In some implementations, the systems and methods describedin this document may reduce resources used by a client device duringhotword analysis with a first, lower threshold, improve an accuracy ofhotword analysis by using a second, more restrictive threshold at aspeech recognition system, or both. In some implementations, the systemsand methods described below may more accurately parse, segment, or both,text in an audio signal, e.g., may more accurately identify a key phraseencoded in the audio signal separate from other utterances encoded inthe audio signal, by sending an entire audio signal, that includes thekey phrase, to a speech recognition system for analysis. In someimplementations, the systems and methods described below may reduceclient processing time, send an audio signal to a speech recognitionsystem more quickly, or both, compared to other systems, when the clientuses a lower hotword analysis threshold than a more restrictive hotwordanalysis threshold used by the speech recognition system.

The details of one or more implementations of the subject matterdescribed in this specification are set forth in the accompanyingdrawings and the description below. Other features, aspects, andadvantages of the subject matter will become apparent from thedescription, the drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an example of an environment in which a client device analyzesan audio signal using a first threshold and a speech recognition systemanalyzes the audio signal using a second threshold that is morerestrictive than the first threshold.

FIG. 2 is a flow diagram of a process for determining whether to performan action.

FIG. 3 is a flow diagram of a process for generating tagged text datafor an audio signal.

FIG. 4 is a block diagram of a computing system that can be used inconnection with computer-implemented methods described in this document.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

FIG. 1 is an example of an environment 100 in which a client device 102analyzes an audio signal using a first threshold and a speechrecognition system 112 analyzes the audio signal using a secondthreshold that is more restrictive than the first threshold. The clientdevice 102 uses the first threshold to determine whether the audiosignal encodes at least a portion of a key phrase. When the clientdevice 102 determines that the audio signal satisfies the firstthreshold of being the key phrase, the client device 102 sends the audiosignal to the speech recognition system 112 that uses the secondthreshold to determine whether the audio signal encodes the entire keyphrase. If so, the speech recognition system 112 may provide the clientdevice 102 with tagged text data of the utterances encoded in the audiosignal to allow the client device 102 to perform an action based on theaudio signal.

The client device 102 includes a microphone 104 that captures the audiosignal. For instance, the client device 102 may be in a lower poweredstate, e.g., standby, while the microphone 104 captures at least part ofthe audio signal. The at least part of the audio signal may be theentire audio signal, one or more first utterances included in the audiosignal, or a different part of the beginning of the audio signal. Oneexample of utterances encoded in an audio signal is “ok google play somemusic.” In this example, the first utterances may be “ok” or “okgoogle.”

The microphone 104 provides the audio signal, or some of the audiosignal as the audio signal is captured, to a client hotword detectionmodule 106. For example, the microphone 104, or a combination ofcomponents in the client device 102, may provide portions of the audiosignal to the client hotword detection module 106 as the audio signal iscaptured by the microphone 104.

The client hotword detection module 106, at time T_(A), determineswhether the audio signal satisfies a first threshold 108. For instance,the client hotword detection module 106 may analyze at least a portionof the one or more first utterances, included at the beginning of theaudio signal, to determine whether the portion of the one or more firstutterances satisfy the first threshold 108 of being a key phrase. Theportion of the first utterances may by “ok” or “ok google.” One exampleof a key phrase may be “ok google.” In some examples, the client hotworddetection module 106 is configured to detect occurrence of only one keyphrase. In some implementations, the client hotword detection module isconfigured to detect occurrence of any of multiple different keyphrases, e.g., ten key phrases. The multiple different key phrasesinclude a limited number of different key phrases for which the clienthotword detection module 106 is trained.

The client hotword detection module 106 may determine a likelihood thatat least a portion of the first utterances are the same as at least aportion of the key phrase. The client hotword detection module 106 maycompare the likelihood with the first threshold 108. When the likelihoodsatisfies the first threshold 108, the client device 102 may send theaudio signal to the speech recognition system 112. When the likelihooddoes not satisfy the first threshold 108, the client device 102 may takeno further action based on the utterances included in the audio signal,e.g., and may discard the audio signal.

The client hotword detection module 106 may determine that the keyphrase is “ok google” and that the utterance “ok”, as one of the firstutterances in the audio signal, satisfies the first threshold 108 ofbeing part of the key phrase. In some examples, the client hotworddetection module 106 may determine that the utterance “ok google” fromthe audio signal satisfies the first threshold 108 of being part of thekey phrase, e.g., the entire key phrase.

In some implementations, when the client hotword detection module 106determines that one or a portion of one of the first utterancessatisfies the first threshold 108 of being a portion of the key phrase,the client hotword detection module 106 may determine whether a totallength of the first utterances matches a length for the key phrases. Forinstance, the client hotword detection module 106 may determine that atime during which the one or more first utterances were spoken matchesan average time for the key phrase to be spoken. The average time may befor a user of the client device 102 or for multiple different people,e.g., including the user of the client device 102.

In some implementations, the client hotword detection module 106 maydetermine that the total length of the first utterances and a totalnumber of n-grams, e.g., words, included in the first utterances matchesa total length of the key phrase and a number of n-grams included in thekey phrase, e.g., when only analyzing a portion of a first utterance orof the first utterances. For instance, the client hotword detectionmodule 106 may determine a number of silences between the firstutterances that indicates the number of first utterances. The clienthotword detection module 106 may compare the number of first utterances,the spoken length of the first utterances, or both, with a total numberof words in the key phrase, the spoken length of the key phrase, orboth. When the client hotword detection module 106 determines that thetotal number of first utterances and the total number of words in thekey phrase are the same, that the spoken length of the first utterancesis within a threshold amount from the spoken length of the key phrase,or both, the client hotword detection module 106 may determine that thefirst utterances in the audio signal satisfy the first threshold 108 ofbeing the key phrase, e.g., when at least a portion of the firstutterances satisfy the first threshold 108 of being a portion of the keyphrase.

The first utterances may satisfy the first threshold 108 of being a keyphrase when the likelihood is greater than the first threshold 108. Thefirst utterances may satisfy the first threshold 108 of being a keyphrase when the likelihood is greater than or equal to the firstthreshold 108. In some examples, the first utterances do not satisfy thefirst threshold 108 of being a key phrase when the likelihood is lessthan the first threshold 108. The first utterances might not satisfy thefirst threshold 108 of being a key phrase when the likelihood is lessthan or equal to the first threshold 108.

In response to determining that at least a portion of the firstutterances satisfy the first threshold 108 of being at least a portionof a key phrase, the client device 102, at time T_(B), sends the audiosignal to the speech recognition system 112. The speech recognitionsystem 112 receives the audio signal and uses a server hotword detectionmodule 114 to determine, at time T_(C), whether the audio signalsatisfies a second threshold 116 of being the key phrase. For instance,the speech recognition system 112 uses the server hotword detectionmodule 114 to determine whether the audio signal satisfies the secondthreshold 116 of being a key phrase.

The second threshold 116 is more restrictive than the first threshold108. For example, the server hotword detection module 114, using thesecond threshold 116, is less likely to incorrectly determine that thefirst utterances represent the same text as a key phrase, e.g., are afalse positive, compared to the client hotword detection module 106,using the first threshold 108. In some examples, when the thresholds arelikelihoods, the first threshold 108 has a lower numerical value thanthe second threshold 116.

The server hotword detection module 114 may use a language model 118, anacoustic model 120, or both, to determine whether the one or more firstutterances satisfy the second threshold 116 of being a key phrase. Thelanguage model 118, and the acoustic model 120, are each trained using alarge amount of training data, e.g., compared to the client hotworddetection module 106. For example, the language model 118, the acousticmodel 120, or both, may be trained using 30,000 hours of training data.The client hotword detection module 106 may be trained using 100 hoursof training data.

In some examples, the server hotword detection module 114 may create ahotword biasing model, that includes the language model 118, theacoustic model 120, or both, on the fly, for use analyzing the audiosignal. For instance, the server hotword detection module 114 mayreceive data from the client device 102 that identifies a key phrase forwhich the server hotword detection module 114 will analyze the audiosignal to determine whether the client device 102 should wake up,perform an action, or both. The data that identifies the key phrase maybe text data for the key phrase, e.g., a text string, or an identifierfor the client device 102, e.g., either of which may be included in therequest to analyze the audio signal received from the client device 102.The server hotword detection module 114 may use the identifier for theclient device 102 to access a database and determine the key phrase forthe client device 102 and the audio signal.

In some examples, the server hotword detection module 114 may use apre-built hotword biasing model. For instance, the server hotworddetection module 114 may analyzes multiple audio signals from the clientdevice 102 or from multiple different client devices, all of which arefor the same key phrase, using the same hotword biasing model.

The hotword biasing model may identify one or more n-grams for which thehotword biasing model performs analysis. For instance, when the keyphrase is “ok google,” the hotword biasing model may generate scores forone or more of the n-grams “<S> ok google,” “<S> ok.” or “ok google,”where <S denotes silence at the beginning of a sentence.

One or both of the language model 118 or the acoustic model 120 may usethe n-grams for the hotword biasing model to determine whether the audiosignal includes the key phrase. For instance, the language model 118 mayuse one or more of the n-grams to generate a score that indicates alikelihood that the audio signal includes the key phrase. The languagemodel 118 may use the n-grams or some of the n-grams to increase alikelihood that the key phrase is correctly identified in the audiosignal when the one or more first utterances are the same as the keyphrase. For example, when the key phrase includes two or more words, thelanguage model 118 may add the key phrase, e.g., “ok google,” to thelanguage model 118 to increase the likelihood that the key phrase isidentified, e.g., compared to when the language model 118 does notalready include the key phrase.

The acoustic model 120 may use one or more of the n-grams to generate ascore that indicates a likelihood that the audio signal includes the keyphrase. For example, the acoustic model 120 may generate multiple scoresfor different phrases, including the key phrase, and select the scorefor the key phrase as output.

The server hotword detection module 114 may receive the two scores fromthe language model 118 and the acoustic model 120. The server hotworddetection module 114 may combine the two scores to determine an overallscore for the audio signal. The server hotword detection module 114 maycompare the overall score with the second threshold 116. When theoverall score satisfies the second threshold 116, the server hotworddetection module 114 determines that the audio signal likely encodes thekey phrase. When the overall score does not satisfy the second threshold116, the server hotword detection module 114 determines that the audiosignal likely does not encode the key phrase.

In response to determining that the audio signal likely does not encodethe key phrase, the speech recognition system 112 may send a message tothe client device 102 indicating that the audio signal does not likelyencode the key phrase. In some examples, the speech recognition system112 might not send the client device 102 a message upon determining thatthe audio signal likely does not encode the key phrase.

When the server hotword detection module 114 determines that the audiosignal likely encodes the key phrase, a tagged text generator 122generates tagged text for the audio signal. The tagged text generator122 may receive data from the language model 118, the acoustic model120, or both, that indicates the n-grams encoded in the audio signal.For instance, the tagged text generator 122 may receive data from theacoustic model 120 that indicates scores for n-grams that are likelyencoded in the audio signal, data representing the n-grams that areencoded in the audio signal, or other appropriate data.

The tagged text generator 122 uses the data from the language model 118,the acoustic model 120, or both, to generate tags for the n-gramsencoded in the audio signal. For example, when the audio signal encodes“ok google play some music,” the tagged text generator 122 may generatedata representing the string “<hotword biasing> ok google </hotwordbiasing> play some music”.

The speech recognition system 112 provides the tagged text for the audiosignal to the client device 102 at time T_(D). The client device 102receives the tagged text and analyzes the tagged text to determine anaction to perform. For instance, the client device 102 may use the tagsincluded in the text to determine which portion of the text correspondsto the key phrase, e.g., the one or more first utterances, and whichportion of the text corresponds to an action for the client device 102to perform. For example, the client device 102 may determine, using thetext “play some music,” to launch a music player application and playmusic. The client device 102 may provide a user prompt requesting inputof a music genre, a music station, an artist, or another type of musicfor playback using the music player application.

In some implementations, the client device 102 may be configured todetect any of multiple different key phrases encoded in an audio signal.For example, the client device 102 may receive input representing a userspecified hotword, such as “hey indigo” or “hey gennie.” The clientdevice 102 may provide the speech recognition system 112 with datarepresenting the user specified hotword. For instance, the client device102 may send a text representation of the user specified hotword withthe audio signal. In some examples, the client device 102 may providethe speech recognition system 112 with data for the user specifiedhotword that the speech recognition system 112 associates with anidentifier for the client device 102, e.g., with a user account for theclient device 102.

The client device 102 may have different key phrases for differentphysical geographic locations. For instance, the client device 102 mayhave a first key phrase for a user's home and a second, different keyphrase for the user's office. The client device 102 may use one or morelocation devices 110 to determine a current physical geographic locationfor the client device 102 and select a corresponding key phrase. Theclient device 102 may send data to the speech recognition system 112with the audio signal that identifies the key phrase based on thephysical geographic location of the client device 102. The locationdevices 110 may include one or more of a global positioning system, awireless device that detects a wireless signature, e.g., of a wirelesshotspot or another device that broadcasts a signature, or a cellularantenna that detects information of cellular base stations.

In some examples, the client device 102 may send data to the speechrecognition system 112 that indicates the physical geographic locationof the client device 102. For instance, the client hotword detectionmodule 106 may be configured for multiple, e.g., five, different keyphrases each of which begin with the same n-gram prefix, e.g., “ok,” andeach of which is for use in a different physical geographic location.For example, the client device 102 may have a key phrase of“ok google”in a first location and “ok indigo” in a second location that is adifferent location from the first location. The client hotword detectionmodule 106 may determine that an audio signal includes the n-gram prefixwithout determining which of the multiple different key phrases may beencoded in the audio signal. The client device 102, upon a determinationby the client hotword detection module 106 that utterances in the audiosignal satisfy the first threshold 108 of being a key phrase, may sendthe audio signal and location data for the client device 102 to thespeech recognition system 112. The speech recognition system 112receives the audio signal and the location data and uses the locationdata to determine a key phrase from the multiple different key phrasesto use for analysis. The server hotword detection module 114 uses thedetermined key phrase to analyze the audio signal and determines whetherthe audio signal satisfies the second threshold 116 of being thedetermined key phrase.

In some implementations, the client device 102 is asleep, e.g., in a lowpower mode, when the client device 102 captures the audio signal, e.g.,using the microphone 104. In the sleep mode, the client device 102 maynot have full functionality. For instance, some features of the clientdevice 102 may be disabled to reduce battery usage.

The client device 102 may begin to wake up upon determining that thefirst utterances satisfy the first threshold 108 of being a key phrase.For example, the client device 102 may enable one or more networkconnectivity devices, one or more of the location devices 110, or both,to allow the client device 102 to communicate with the speechrecognition system 112.

When the client device 102 receives the tagged text data from the speechrecognition system 112, the client device 102 exits the sleep mode. Forinstance, the client device 102 enables more functionality of the clientdevice 102 to determine an action to perform using the tagged text, toperform an action determined using the tagged text, or both.

The speech recognition system 112 is an example of a system implementedas computer programs on one or more computers in one or more locations,in which the systems, components, and techniques described in thisdocument are implemented. The client device 102 may include a personalcomputer, a mobile communication device, or another device that can sendand receive data over a network 124. The network 124, such as a localarea network (LAN), wide area network (WAN), the Internet, or acombination thereof, connects the client device 102, and the speechrecognition system 112. The speech recognition system 112 may use asingle server computer or multiple server computers operating inconjunction with one another, including, for example, a set of remotecomputers deployed as a cloud computing service.

FIG. 2 is a flow diagram of a process 200 for determining whether toperform an action. For example, the process 200 can be used by theclient device 102 from the environment 100.

A client device receives an audio signal encoding one or more utterancesincluding a first utterance (202). The client device may use anyappropriate type of device to capture the audio signal. In someexamples, the client device may receive the audio signal from anotherdevice, e.g., a smart watch.

The client device determines whether at least a portion of the firstutterance satisfies a first threshold of being at least a portion of akey phrase (204). The client device may include data for one or more keyphrases. The client device may determine whether at least the portion ofthe first utterance has at least a predetermined likelihood, defined bythe first threshold, of being a portion of one of the key phrases. Theportion of the first utterance may include one or more n-grams from thefirst utterance or another appropriate type of segment from the firstutterance. In some examples, when the key phrase includes two or morewords, the portion may include a single word from two or more firstutterances. In some examples, the client device may determine whethermultiple first utterances, e.g., one or more first utterances, satisfythe first threshold of being one of the key phrases.

In response to determining that at least a portion of the firstutterance satisfies the first threshold of being at least a portion of akey phrase, the client device sends the audio signal to a server systemthat determines whether the first utterance satisfies a second thresholdof being the key phrase (206). The second threshold is more restrictivethan the first threshold. For instance, the client device may send theaudio signal, or a portion of the audio signal, to the server, e.g., aspeech recognition system, to cause the server to determine whether thefirst utterance satisfies the second threshold of being the key phrase.The server always analyzes all of the first utterances to determinewhether the first utterances satisfy the second threshold of being theentire key phrase.

The client device determines whether response data, received from theserver system, includes tagged text data representing the one or moreutterances encoded in the audio signal (208). For example, the clientdevice may receive the response data from the server in response tosending the audio signal to the server. The client device may analyzethe response data to determine whether the response data includes taggedtext data.

In response to determining that the response data includes tagged textdata representing the one or more utterances encoded in the audiosignal, the client device performs an action using the tagged text data(210). For instance, the client device uses the tags in the data todetermine the action to perform. The tags may indicate which portion ofthe tagged data, and the respective portion of the audio signal,correspond to the first utterances for the key phrase. The tags mayindicate which portion of the tagged data correspond to an action forthe client device to perform, e.g., “play some music.”

In response to determining that at least a portion of the firstutterance does not satisfy the first threshold of being at least aportion of a key phrase or in response to determining that the responsedata does not include tagged text data, the client device determines tonot perform an action using data from the audio signal (212). Forinstance, when none of the first utterance satisfies the first thresholdof being the key phrase, the client device does not perform any actionusing the audio signal. In some examples, when the client devicereceives a message from the server that indicates that the audio signaldid not encode the key phrase, e.g., the response data does not includetagged text data, the client device does not perform any action usingthe audio signal.

In response to determining that at least a portion of the firstutterance does not satisfy the first threshold of being at least aportion of a key phrase or in response to determining that the responsedata does not include tagged text data, the client device discards theaudio signal (214). For instance, when none of the first utterancesatisfies the first threshold of being the key phrase, the client devicemay discard the audio signal. In some examples, when the client devicereceives a message from the server that indicates that the audio signaldid not encode the key phrase, e.g., the response data does not includetagged text data, the client device may discard the audio signal. Insome implementations, the client device may discard the audio signalafter a predetermined period of time when one of these conditionsoccurs.

The order of steps in the process 200 described above is illustrativeonly, and determining whether to perform an action can be performedindifferent orders. For example, the client device may discard the audiosignal and then not perform an action using data from the audio signalor may perform these two steps concurrently.

In some implementations, the process 200 can include additional steps,fewer steps, or some of the steps can be divided into multiple steps.For example, the client device may either discard the audio signal ornot perform an action using data from the audio signal, instead ofperforming both steps.

FIG. 3 is a flow diagram of a process 300 for generating tagged textdata for an audio signal. For example, the process 300 can be used bythe speech recognition system 112 from the environment 100.

A speech recognition system receives, from a client device, an audiosignal encoding one or more utterances including one or more firstutterances for which the client device determined that at least aportion of the first utterance satisfies a first threshold of being atleast a portion of a key phrase (302). The speech recognition system mayreceive the audio signal from the client device across a network. Theclient device may have sent the audio signal to the speech recognitionsystem as part of a process that includes performing steps 202 through206 described above with reference to FIG. 2.

The speech recognition system customizes a language model for the keyphrase (304). For instance, the speech recognition system may increase alikelihood that the language model, which is not specific to anyparticular key phrase, will accurately identify an occurrence of the keyphrase encoded in the audio signal. In some examples, the speechrecognition system may adjust weights for the language model specific tothe key phrase.

In some implementations, the speech recognition system may determinewhether to use a default key phrase. For instance, the speechrecognition system may determine whether a message received from theclient device that includes the audio signal also includes dataidentifying a key phrase, e.g., text for the key phrase or an identifierthat can be used to look up a key phrase in a database. The speechrecognition system may determine to use a default key phrase when themessage does not include data identifying a key phrase. For example, thespeech recognition system may determine that the client device, or acorresponding user account, does not have a customized key phrase and touse a default key phrase.

The speech recognition system determines whether the one or more firstutterances satisfy the second threshold of being a key phrase based onoutput from the language model, an acoustic model, or both (306). Forinstance, the speech recognition system provides the audio signal to thelanguage model, the acoustic model, or both. The speech recognitionsystem receives a score from the language model, the acoustic model, orboth, that each indicate a likelihood that the one or more firstutterances are the key phrase. The speech recognition system may combinethe separate scores from the language model and the acoustic model todetermine whether the combined score for the audio signal satisfies thesecond threshold of being the key phrase.

In response to determining that the first utterance satisfies the secondthreshold of being a key phrase based on output from the language model,the acoustic model, or both, the speech recognition system analyzes theentire audio signal to determine data for each of the one or moreutterances (308). For example, an acoustic model generates outputindicating a text string for the words likely encoded in the audiosignal. A tagged text generator may apply tags to the text string thatindicate one or more attributes of n-grams, e.g., words, included in thetext sting. For instance, the tagged text generator may apply tags thatidentify a key phrase, an action word, e.g., “play,” an application,e.g., music player, or a combination of two or more of these, to thetext string.

The speech recognition system sends, to the client device, tagged textdata representing the one or more utterances encoded in the audio signalgenerated using the data for each of the one or more utterances (310).The speech recognition system may send the tagged text data to theclient device to cause the client device to perform an action using thetagged text data.

In response to determining that the first utterance does not satisfy thesecond threshold of being a key phrase based on output from the languagemodel, the acoustic model, or both, the speech recognition system sends,to the client device, data indicating that the key phrase is not likelyencoded in the audio signal (312). For instance, the speech recognitionsystem may provide the client device with a message that indicates thatthe client device should not perform any action using data for the audiosignal.

In some implementations, the process 300 can include additional steps,fewer steps, or some of the steps can be divided into multiple steps.For example, the speech recognition system might not customize thelanguage model. In some examples, the speech recognition system maydetermine whether the first utterance satisfies the second threshold ofbeing a key phrase using data or systems other than the language model,the acoustic model, or both.

Embodiments of the subject matter and the functional operationsdescribed in this specification can be implemented in digital electroniccircuitry, in tangibly-embodied computer software or firmware, incomputer hardware, including the structures disclosed in thisspecification and their structural equivalents, or in combinations ofone or more of them. Embodiments of the subject matter described in thisspecification can be implemented as one or more computer programs, i.e.,one or more modules of computer program instructions encoded on atangible non-transitory program carrier for execution by, or to controlthe operation of, data processing apparatus. Alternatively or inaddition, the program instructions can be encoded on anartificially-generated propagated signal, e.g., a machine-generatedelectrical, optical, or electromagnetic signal, that is generated toencode information for transmission to suitable receiver apparatus forexecution by a data processing apparatus. The computer storage mediumcan be a machine-readable storage device, a machine-readable storagesubstrate, a random or serial access memory device, or a combination ofone or more of them.

The term “data processing apparatus” refers to data processing hardwareand encompasses all kinds of apparatus, devices, and machines forprocessing data, including by way of example a programmable processor, acomputer, or multiple processors or computers. The apparatus can also beor further include special purpose logic circuitry, e.g., an FPGA (fieldprogrammable gate array) or an ASIC (application-specific integratedcircuit). The apparatus can optionally include, in addition to hardware,code that creates an execution environment for computer programs, e.g.,code that constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, or a combination of one or moreof them.

A computer program, which may also be referred to or described as aprogram, software, a software application, a module, a software module,a script, or code, can be written in any form of programming language,including compiled or interpreted languages, or declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, or other unitsuitable for use in a computing environment. A computer program may, butneed not, correspond to a file in a file system. A program can be storedin a portion of a file that holds other programs or data, e.g., one ormore scripts stored in a markup language document, in a single filededicated to the program in question, or in multiple coordinated files,e.g., files that store one or more modules, sub-programs, or portions ofcode. A computer program can be deployed to be executed on one computeror on multiple computers that are located at one site or distributedacross multiple sites and interconnected by a communication network.

The processes and logic flows described in this specification can beperformed by one or more programmable computers executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application-specific integrated circuit).

Computers suitable for the execution of a computer program include, byway of example, general or special purpose microprocessors or both, orany other kind of central processing unit. Generally, a centralprocessing unit will receive instructions and data from a read-onlymemory or a random access memory or both. The essential elements of acomputer are a central processing unit for performing or executinginstructions and one or more memory devices for storing instructions anddata. Generally, a computer will also include, or be operatively coupledto receive data from or transfer data to, or both, one or more massstorage devices for storing data, e.g., magnetic, magneto-optical disks,or optical disks. However, a computer need not have such devices.Moreover, a computer can be embedded in another device, e.g., a mobiletelephone, a smart phone, a personal digital assistant (PDA), a mobileaudio or video player, a game console, a Global Positioning System (GPS)receiver, or a portable storage device, e.g., a universal serial bus(USB) flash drive, to name just a few.

Computer-readable media suitable for storing computer programinstructions and data include all forms of non-volatile memory, mediaand memory devices, including by way of example semiconductor memorydevices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks,e.g., internal hard disks or removable disks; magneto-optical disks; andCD-ROM and DVD-ROM disks. The processor and the memory can besupplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, embodiments of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., LCD (liquid crystal display), OLED(organic light emitting diode) or other monitor, for displayinginformation to the user and a keyboard and a pointing device, e.g., amouse or a trackball, by which the user can provide input to thecomputer. Other kinds of devices can be used to provide for interactionwith a user as well; for example, feedback provided to the user can beany form of sensory feedback, e.g., visual feedback, auditory feedback,or tactile feedback; and input from the user can be received in anyform, including acoustic, speech, or tactile input. In addition, acomputer can interact with a user by sending documents to and receivingdocuments from a device that is used by the user; for example, bysending web pages to a web browser on a user's device in response torequests received from the web browser.

Embodiments of the subject matter described in this specification can beimplemented in a computing system that includes a back-end component,e.g., as a data server, or that includes a middleware component, e.g.,an application server, or that includes a front-end component, e.g., aclient computer having a graphical user interface or a Web browserthrough which a user can interact with an implementation of the subjectmatter described in this specification, or any combination of one ormore such back-end, middleware, or front-end components. The componentsof the system can be interconnected by any form or medium of digitaldata communication, e.g., a communication network Examples ofcommunication networks include a local area network (LAN) and a widearea network (WAN), e.g., the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other. In someembodiments, a server transmits data, e.g., an HyperText Markup Language(HTML) page, to a user device, e.g., for purposes of displaying data toand receiving user input from a user interacting with the user device,which acts as a client. Data generated at the user device, e.g., aresult of the user interaction, can be received from the user device atthe server.

FIG. 4 is a block diagram of computing devices 400, 450 that may be usedto implement the systems and methods described in this document, aseither a client or as a server or plurality of servers. Computing device400 is intended to represent various forms of digital computers, such aslaptops, desktops, workstations, personal digital assistants, servers,blade servers, mainframes, and other appropriate computers. Computingdevice 450 is intended to represent various forms of mobile devices,such as personal digital assistants, cellular telephones, smartphones,smartwatches, head-worn devices, and other similar computing devices.The components shown here, their connections and relationships, andtheir functions, are meant to be exemplary only, and are not meant tolimit implementations described and/or claimed in this document.

Computing device 400 includes a processor 402, memory 404, a storagedevice 406, a high-speed interface 408 connecting to memory 404 andhigh-speed expansion ports 410, and a low speed interface 412 connectingto low speed bus 414 and storage device 406. Each of the components 402,404, 406, 408, 410, and 412, are interconnected using various busses,and may be mounted on a common motherboard or in other manners asappropriate. The processor 402 can process instructions for executionwithin the computing device 400, including instructions stored in thememory 404 or on the storage device 406 to display graphical informationfor a GUI on an external input/output device, such as display 416coupled to high speed interface 408. In other implementations, multipleprocessors and/or multiple buses may be used, as appropriate, along withmultiple memories and types of memory. Also, multiple computing devices400 may be connected, with each device providing portions of thenecessary operations (e.g., as a server bank, a group of blade servers,or a multi-processor system).

The memory 404 stores information within the computing device 400. Inone implementation, the memory 404 is a computer-readable medium. In oneimplementation, the memory 404 is a volatile memory unit or units. Inanother implementation, the memory 404 is a non-volatile memory unit orunits.

The storage device 406 is capable of providing mass storage for thecomputing device 400. In one implementation, the storage device 406 is acomputer-readable medium. In various different implementations, thestorage device 406 may be a floppy disk device, a hard disk device, anoptical disk device, or a tape device, a flash memory or other similarsolid state memory device, or an array of devices, including devices ina storage area network or other configurations. In one implementation, acomputer program product is tangibly embodied in an information carrier.The computer program product contains instructions that, when executed,perform one or more methods, such as those described above. Theinformation carrier is a computer- or machine-readable medium, such asthe memory 404, the storage device 406, or memory on processor 402.

The high speed controller 408 manages bandwidth-intensive operations forthe computing device 400, while the low speed controller 412 manageslower bandwidth-intensive operations. Such allocation of duties isexemplary only. In one implementation, the high-speed controller 408 iscoupled to memory 404, display 416 (e.g., through a graphics processoror accelerator), and to high-speed expansion ports 410, which may acceptvarious expansion cards (not shown). In the implementation, low-speedcontroller 412 is coupled to storage device 406 and low-speed expansionport 414. The low-speed expansion port, which may include variouscommunication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet)may be coupled to one or more input/output devices, such as a keyboard,a pointing device, a scanner, or a networking device such as a switch orrouter, e.g., through a network adapter.

The computing device 400 may be implemented in a number of differentforms, as shown in the figure. For example, it may be implemented as astandard server 420, or multiple times in a group of such servers. Itmay also be implemented as part of a rack server system 424. Inaddition, it may be implemented in a personal computer such as a laptopcomputer 422. Alternatively, components from computing device 400 may becombined with other components in a mobile device (not shown), such asdevice 450. Each of such devices may contain one or more of computingdevice 400, 450, and an entire system may be made up of multiplecomputing devices 400, 450 communicating with each other.

Computing device 450 includes a processor 452, memory 464, aninput/output device such as a display 454, a communication interface466, and a transceiver 468, among other components. The device 450 mayalso be provided with a storage device, such as a microdrive or otherdevice, to provide additional storage. Each of the components 450, 452,464, 454, 466, and 468, are interconnected using various buses, andseveral of the components may be mounted on a common motherboard or inother manners as appropriate.

The processor 452 can process instructions for execution within thecomputing device 450, including instructions stored in the memory 464.The processor may also include separate analog and digital processors.The processor may provide, for example, for coordination of the othercomponents of the device 450, such as control of user interfaces,applications run by device 450, and wireless communication by device450.

Processor 452 may communicate with a user through control interface 458and display interface 456 coupled to a display 454. The display 454 maybe, for example, a TFT LCD display or an OLED display, or otherappropriate display technology. The display interface 456 may compriseappropriate circuitry for driving the display 454 to present graphicaland other information to a user. The control interface 458 may receivecommands from a user and convert them for submission to the processor452. In addition, an external interface 462 may be provide incommunication with processor 452, so as to enable near areacommunication of device 450 with other devices. External interface 462may provide, for example, for wired communication (e.g., via a dockingprocedure) or for wireless communication (e.g., via Bluetooth or othersuch technologies).

The memory 464 stores information within the computing device 450. Inone implementation, the memory 464 is a computer-readable medium. In oneimplementation, the memory 464 is a volatile memory unit or units. Inanother implementation, the memory 464 is a non-volatile memory unit orunits. Expansion memory 474 may also be provided and connected to device450 through expansion interface 472, which may include, for example, aSIMM card interface. Such expansion memory 474 may provide extra storagespace for device 450, or may also store applications or otherinformation for device 450. Specifically, expansion memory 474 mayinclude instructions to carry out or supplement the processes describedabove, and may include secure information also. Thus, for example,expansion memory 474 may be provide as a security module for device 450,and may be programmed with instructions that permit secure use of device450. In addition, secure applications may be provided via the SIMMcards, along with additional information, such as placing identifyinginformation on the SIMM card in a non-hackable manner.

The memory may include for example, flash memory and/or MRAM memory, asdiscussed below. In one implementation, a computer program product istangibly embodied in an information carrier. The computer programproduct contains instructions that, when executed, perform one or moremethods, such as those described above. The information carrier is acomputer- or machine-readable medium, such as the memory 464, expansionmemory 474, or memory on processor 452.

Device 450 may communicate wirelessly through communication interface466, which may include digital signal processing circuitry wherenecessary. Communication interface 466 may provide for communicationsunder various modes or protocols, such as GSM voice calls, SMS, EMS, orMMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others.Such communication may occur, for example, through radio-frequencytransceiver 468. In addition, short-range communication may occur, suchas using a Bluetooth, WiFi, or other such transceiver (not shown). Inaddition, GPS receiver module 470 may provide additional wireless datato device 450, which may be used as appropriate by applications runningon device 450.

Device 450 may also communicate audibly using audio codec 460, which mayreceive spoken information from a user and convert it to usable digitalinformation. Audio codec 460 may likewise generate audible sound for auser, such as through a speaker, e.g., in a handset of device 450. Suchsound may include sound from voice telephone calls, may include recordedsound (e.g., voice messages, music files, etc.) and may also includesound generated by applications operating on device 450.

The computing device 450 may be implemented in a number of differentforms, as shown in the figure. For example, it may be implemented as acellular telephone 480. It may also be implemented as part of asmartphone 482, personal digital assistant, or other similar mobiledevice.

Various implementations of the systems and techniques described here canbe realized in digital electronic circuitry, integrated circuitry,specially designed ASICs (application specific integrated circuits),computer hardware, firmware, software, and/or combinations thereof.These various implementations can include implementation in one or morecomputer programs that are executable and/or interpretable on aprogrammable system including at least one programmable processor, whichmay be special or general purpose, coupled to receive data andinstructions from, and to transmit data and instructions to, a storagesystem, at least one input device, and at least one output device.

These computer programs (also known as programs, software, softwareapplications or code) include machine instructions for a programmableprocessor, and can be implemented in a high-level procedural and/orobject-oriented programming language, and/or in assembly/machinelanguage. As used herein, the terms “machine-readable medium”“computer-readable medium” refers to any computer program product,apparatus and/or device (e.g., magnetic discs, optical disks, memory.Programmable Logic Devices (PLDs)) used to provide machine instructionsand/or data to a programmable processor, including a machine-readablemedium that receives machine instructions as a machine-readable signal.The term “machine-readable signal” refers to any signal used to providemachine instructions and/or data to a programmable processor.

To provide for interaction with a user, the systems and techniquesdescribed here can be implemented on a computer having a display device(e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor)for displaying information to the user and a keyboard and a pointingdevice (e.g., a mouse or a trackball) by which the user can provideinput to the computer. Other kinds of devices can be used to provide forinteraction with a user as well; for example, feedback provided to theuser can be any form of sensory feedback (e.g., visual feedback,auditory feedback, or tactile feedback); and input from the user can bereceived in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in acomputing system that includes a back-end component (e.g., as a dataserver), or that includes a middleware component (e.g., an applicationserver), or that includes a front-end component (e.g., a client computerhaving a graphical user interface or a Web browser through which a usercan interact with an implementation of the systems and techniquesdescribed here), or any combination of such back-end, middleware, orfront-end components. The components of the system can be interconnectedby any form or medium of digital data communication (e.g., acommunication network). Examples of communication networks include alocal area network (“LAN”), a wide area network (“WAN”), and theInternet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of what may beclaimed, but rather as descriptions of features that may be specific toparticular embodiments. Certain features that are described in thisspecification in the context of separate embodiments can also beimplemented in combination in a single embodiment. Conversely, variousfeatures that are described in the context of a single embodiment canalso be implemented in multiple embodiments separately or in anysuitable subcombination. Moreover, although features may be describedabove as acting in certain combinations and even initially claimed assuch, one or more features from a claimed combination can in some casesbe excised from the combination, and the claimed combination may bedirected to a subcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various system modulesand components in the embodiments described above should not beunderstood as requiring such separation in all embodiments, and itshould be understood that the described program components and systemscan generally be integrated together in a single software product orpackaged into multiple software products.

Particular embodiments of the subject matter have been described. Otherembodiments are within the scope of the following claims. For example,the actions recited in the claims can be performed in a different orderand still achieve desirable results. As one example, the processesdepicted in the accompanying figures do not necessarily require theparticular order shown, or sequential order, to achieve desirableresults. In some cases, multitasking and parallel processing may beadvantageous.

What is claimed is:
 1. A device-implemented method, comprising:receiving, by a device, an audio signal encoding one or more utterancesincluding a first utterance; determining, by the device, whether atleast a portion of the first utterance satisfies a first threshold ofbeing at least a portion of a key phrase; in response to determiningthat at least the portion of the first utterance satisfies the firstthreshold of being at least the portion of the key phrase, sending, bythe device, the received audio signal to a server that determineswhether the first utterance satisfies a second threshold of being thekey phrase, the second threshold being more restrictive than the firstthreshold; and receiving, by the device and from the server, data thatindicates the first utterance satisfies the second threshold.
 2. Themethod of claim 1, wherein the sending the audio signal by the deviceincludes sending an indication of the key phrase from among multipledifferent key phrases.
 3. The method of claim 2, wherein the sending theindication of the key phrase from among multiple different key phrasescomprises sending a textual representation of the key phrase.
 4. Themethod of claim 1, wherein the first threshold corresponds to alikelihood of the first utterance being at least the portion of the keyphrase that is greater than a likelihood of the first utterance being atleast the portion of the key phrase that corresponds to the secondthreshold.
 5. The method of claim 1, wherein the data that indicates thefirst utterance satisfies the second threshold comprises data thatindicates the first utterance includes the key phrase.
 6. The method ofclaim 1, wherein the determining whether at least the portion of thefirst utterance satisfies the first threshold of being at least theportion of the key phrase causes the device to wake up.
 7. The method ofclaim 1, wherein the determining whether at least the portion of thefirst utterance satisfies the first threshold of being at least theportion of the key phrase comprises: for each of a set of multipledifferent key phrases, determining whether at least the portion of thefirst utterance satisfies the first threshold of being at least theportion of the key phrase of the set of multiple different key phrases.8. A device, comprising: processing circuitry configured to receive anaudio signal encoding one or more utterances including a firstutterance, determine whether at least a portion of the first utterancesatisfies a first threshold of being at least a portion of a key phrase,in response to determining that at least the portion of the firstutterance satisfies the first threshold of being at least the portion ofthe key phrase, send the audio signal to a server that determineswhether the first utterance satisfies a second threshold of being thekey phrase, the second threshold being more restrictive than the firstthreshold, and receive, from the server, data that indicates the firstutterance satisfies the second threshold.
 9. The device of claim 8,wherein the processing circuitry is configured to send, as a componentof the audio signal, an indication of the key phrase from among multipledifferent key phrases.
 10. The device of claim 9, wherein the processingcircuitry is configured to send, as the indication of the key phrasefrom among multiple different key phrases, a textual representation ofthe key phrase.
 11. The device of claim 8, wherein the first thresholdcorresponds to a likelihood of the first utterance being at least theportion of the key phrase that is greater than a likelihood of the firstutterance being at least the portion of the key phrase that correspondsto the second threshold.
 12. The device of claim 8, wherein the datathat indicates the first utterance satisfies the second thresholdcomprises data that indicates the first utterance includes the keyphrase.
 13. The device of claim 8, wherein determining whether at leastthe portion of the first utterance satisfies the first threshold ofbeing at least the portion of the key phrase causes the device to wakeup.
 14. The device of claim 8, wherein the processing circuitry isconfigured to determine whether at least the portion of the firstutterance satisfies the first threshold of being at least the portion ofthe key phrase by: for each of a set of multiple different key phrases,determining whether at least the portion of the first utterancesatisfies the first threshold of being at least the portion of the keyphrase of the set of multiple different key phrases.
 15. Anon-transitory computer-readable medium storing software comprisinginstructions executable by a device which, upon such execution, causethe device to perform operations, comprising: receiving an audio signalencoding one or more utterances including a first utterance; determiningwhether at least a portion of the first utterance satisfies a firstthreshold of being at least a portion of a key phrase; in response todetermining that at least the portion of the first utterance satisfiesthe first threshold of being at least the portion of the key phrase,sending the audio signal to a server that determines whether the firstutterance satisfies a second threshold of being the key phrase, thesecond threshold being more restrictive than the first threshold; andreceiving, from the server, data that indicates the first utterancesatisfies the second threshold.
 16. The medium of claim 15, wherein thesending the audio signal includes sending an indication of the keyphrase from among multiple different key phrases.
 17. The medium ofclaim 16, wherein sending an indication of the key phrase from amongmultiple different key phrases comprises sending a textualrepresentation of the key phrase.
 18. The medium of claim 15, whereinthe first threshold corresponds to a likelihood of the first utterancebeing at least the portion of the key phrase that is greater than alikelihood of the first utterance being at least the portion of the keyphrase that corresponds to the second threshold.
 19. The medium of claim15, wherein the data that indicates that the server determined that thefirst utterance satisfies the second threshold comprises data thatindicates the first utterance includes the key phrase.
 20. The medium ofclaim 15, wherein the determining whether at least the portion of thefirst utterance satisfies the first threshold of being at least theportion of the key phrase causes the device to wake up.